Add new KeySpacePath.exportAllData #3566
Merged
+1,781
−157
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This will be returned by export
Most of the time this will be used with remainders, so make most of the tests cover that
ScottDugas
commented
Sep 8, 2025
| import javax.annotation.Nonnull; | ||
| import java.util.UUID; | ||
|
|
||
| /** |
Collaborator
Author
There was a problem hiding this comment.
In case you're reviewing this top-to-bottom, this class was extracted from KeySpaceDirectoryTest, but setupSampleData was added.
alecgrieser
requested changes
Sep 9, 2025
...main/java/com/apple/foundationdb/record/provider/foundationdb/keyspace/KeySpacePathImpl.java
Show resolved
Hide resolved
...in/java/com/apple/foundationdb/record/provider/foundationdb/keyspace/DataInKeySpacePath.java
Outdated
Show resolved
Hide resolved
...in/java/com/apple/foundationdb/record/provider/foundationdb/keyspace/DataInKeySpacePath.java
Outdated
Show resolved
Hide resolved
...in/java/com/apple/foundationdb/record/provider/foundationdb/keyspace/DataInKeySpacePath.java
Outdated
Show resolved
Hide resolved
...ava/com/apple/foundationdb/record/provider/foundationdb/keyspace/DataInKeySpacePathTest.java
Outdated
Show resolved
Hide resolved
...ava/com/apple/foundationdb/record/provider/foundationdb/keyspace/DataInKeySpacePathTest.java
Outdated
Show resolved
Hide resolved
...in/java/com/apple/foundationdb/record/provider/foundationdb/keyspace/DataInKeySpacePath.java
Show resolved
Hide resolved
...com/apple/foundationdb/record/provider/foundationdb/keyspace/KeySpacePathDataExportTest.java
Outdated
Show resolved
Hide resolved
...com/apple/foundationdb/record/provider/foundationdb/keyspace/KeySpacePathDataExportTest.java
Outdated
Show resolved
Hide resolved
...com/apple/foundationdb/record/provider/foundationdb/keyspace/KeySpacePathDataExportTest.java
Outdated
Show resolved
Hide resolved
ScottDugas
commented
Sep 12, 2025
Collaborator
Author
ScottDugas
left a comment
ScottDugas
left a comment
There was a problem hiding this comment.
I went through a bunch of the comments, and responded. I've made a bunch of the changes locally, but when I got to implementing DataInKeySpacePath.equals/hashCode, I discovered that ResolvedKeySpacePath does not implement those correctly. I'm going to pause, the work on this PR, to resolve that. I'm submitting my commits in the meantime, for visibility.
...main/java/com/apple/foundationdb/record/provider/foundationdb/keyspace/KeySpacePathImpl.java
Show resolved
Hide resolved
...in/java/com/apple/foundationdb/record/provider/foundationdb/keyspace/DataInKeySpacePath.java
Show resolved
Hide resolved
...in/java/com/apple/foundationdb/record/provider/foundationdb/keyspace/DataInKeySpacePath.java
Outdated
Show resolved
Hide resolved
...src/main/java/com/apple/foundationdb/record/provider/foundationdb/keyspace/KeySpacePath.java
Outdated
Show resolved
Hide resolved
...in/java/com/apple/foundationdb/record/provider/foundationdb/keyspace/DataInKeySpacePath.java
Outdated
Show resolved
Hide resolved
...ava/com/apple/foundationdb/record/provider/foundationdb/keyspace/DataInKeySpacePathTest.java
Outdated
Show resolved
Hide resolved
...ava/com/apple/foundationdb/record/provider/foundationdb/keyspace/DataInKeySpacePathTest.java
Show resolved
Hide resolved
...ava/com/apple/foundationdb/record/provider/foundationdb/keyspace/DataInKeySpacePathTest.java
Outdated
Show resolved
Hide resolved
...ava/com/apple/foundationdb/record/provider/foundationdb/keyspace/DataInKeySpacePathTest.java
Outdated
Show resolved
Hide resolved
...in/java/com/apple/foundationdb/record/provider/foundationdb/keyspace/DataInKeySpacePath.java
Show resolved
Hide resolved
I left a TODO for myself to add unit tests of withRemainder...
I updated the tests to not use KeyValue directly, at which point, having DataInKeySpacePath expose the value and not the raw KeyValue seemed to make sense
This isn't the greatest test, but I think the proper test would be to explicitly list the non-default methods and then use reflection and error if any new methods are added and don't have a default implementation.
ohadzeliger
requested changes
Oct 22, 2025
...src/main/java/com/apple/foundationdb/record/provider/foundationdb/keyspace/KeySpacePath.java
Outdated
Show resolved
Hide resolved
...src/main/java/com/apple/foundationdb/record/provider/foundationdb/keyspace/KeySpacePath.java
Show resolved
Hide resolved
...src/main/java/com/apple/foundationdb/record/provider/foundationdb/keyspace/KeySpacePath.java
Show resolved
Hide resolved
.../java/com/apple/foundationdb/record/provider/foundationdb/keyspace/ResolvedKeySpacePath.java
Outdated
Show resolved
Hide resolved
| keySpace.root().getValue(), keySpace.root(), context)); | ||
| } | ||
| } | ||
|
|
Contributor
There was a problem hiding this comment.
Maybe add a test that fails to resolve a key?
Would it make sense to add test for resolve with NULL value?
Collaborator
Author
There was a problem hiding this comment.
I added a test that fails to resolve (has a different root)
Regarding NULL, are you suggesting creating a DataInKeySpacePath with KeyValue(key, null), or a path that has a NULL type in it?
For the former, I don't think FDB will ever return such a KeyValue, and if you have one, you would never be able to insert it, because set does not allow a null value. Give that, it probably makes sense to have DataInKeySpacePath error if it gets a null value.
For the later, I think that should be better tested in KeySpacePath.toResolvedPathAsync.
Contributor
There was a problem hiding this comment.
I meant the later - NULL type.
...com/apple/foundationdb/record/provider/foundationdb/keyspace/KeySpacePathDataExportTest.java
Show resolved
Hide resolved
...com/apple/foundationdb/record/provider/foundationdb/keyspace/KeySpacePathDataExportTest.java
Show resolved
Hide resolved
Predominately: - DataInKeySpace now errors if it gets a null value - Improved javadoc linking KeySpace.resolveFromKeyAsync and KeySpacePath.toResolvedPathAsync - Fix error message in default method - Reduce visibility of ResolvedKeySPacePath.withRemainder - new negative tests in DataInKeySpacePathTest - Better coverage of exporting with mixed-type sub-directories
ohadzeliger
approved these changes
Oct 28, 2025
alecgrieser
approved these changes
Oct 29, 2025
fdb-extensions/src/main/java/com/apple/foundationdb/tuple/TupleHelpers.java
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This introduces a new KeySpacePath.exportAllData to export all the data stored in the path. This can eventually be used to import into another cluster, or back into the same cluster, after clearing.
Other than the path information, the data exported is raw bytes, with no transformation or indicated semantics.
Resolves: 3572